Some Two–Step and Three–Step Nilpotent Lie Groups with Small Automorphism Groups

We construct examples of two-step and three-step nilpotent Lie groups whose automorphism groups are “small” in the sense of either not having a dense orbit for the action on the Lie group, or being nilpotent (the latter being stronger). From the results we also get new examples of compact manifolds covered by two-step simply connected nilpotent Lie groups which do not admit Anosov automorphisms. Let G be a connected Lie group and Aut (G) the group of all (continuous) automorphisms of G. When does the action of Aut (G) on G have a dense orbit? It turns out that if this holds, then G is a nilpotent Lie group (see [4]). However, it does not hold for all nilpotent Lie groups, and a characterization of the class of groups for which it holds seems to be a remote possibility. When G is a vector space Aut (G) is its general linear group, and the action has an open dense orbit, namely the complement of the zero. On the other hand, there exist 3-step simply connected nilpotent Lie groups such that every orbit of the action is a proper closed subset; this holds for the simply connected Lie group corresponding to the Lie algebra described in [9]; see also § 4 below. Among the simply connected Lie groups this brings us to considering the question for 2-step nilpotent Lie groups. In this note, using a representation of PSL(2,R), we construct an example of a 2-step simply connected nilpotent Lie group G for which the action of the automorphism group has no dense orbit; see Theorem 2.1. We discuss the question also for Lie groups which are not simply connected, and show that while for all connected abelian groups other than the circle the automorphism group action has dense orbits, among the quotients of the group G as above by discrete central subgroups there are nilpotent Lie groups G′ such that every orbit of Aut (G′) on G′ consists of either one or two cosets of the commutator subgroup [G′, G′], the latter being a closed subgroup; see Corollary 5.2. The method involved also enables us to give examples of 3-step simply connected nilpotent Lie groups whose automorphism groups are nilpotent and act on the Lie algebras by unipotent transformations; this condition implies in particular that all orbits of the actions are closed; see § 4. An example of a 6-step simply connected nilpotent Lie group with this property was given earlier in [10]. Another motivation for studying the automorphism groups of nilpotent Lie groups comes from the question of understanding which compact nilmanifolds support Anosov diffeomorphisms and which do not. It is known that if G is a free k-step nilpotent Lie group over an n-dimensional vector space V with k < n, then Received by the editors April 29, 2002 and, in revised form, July 12, 2002. 2000 Mathematics Subject Classification. Primary 22D45, 22E25; Secondary 22D40, 37D20. c ©2002 American Mathematical Society